In [18]:
# This needs to be at the top of every notebook
import envbash
#envbash.load_envbash('/home/jovyan/.profile')
In [19]:
# This cell sets up the notebook to use the Go demo account (read-only) token
# *Remove this if you are modifying this notebook for your own projects*
os.environ['GO_API_TOKEN'] = os.environ['GO_DEMO_API_TOKEN']
In [20]:
import os
import numpy as np
import pandas as pd
import datetime
import geoloc_heatmap as geo_map
from go_feature_creation import TemporalFeatures
import geoloc_project_creation as gpc
import geoloc_heatmap as gh
from geoloc_heatmap import *
from demos.tools import go_utils
from demos.tools import aoi_utils
from polygon_geohasher.polygon_geohasher import geohash_to_polygon
import json
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
import geopandas as gpd
from ipyleaflet import Map, GeoData, Marker, basemaps
from ipywidgets import HTML
mapbox_access_token = os.getenv('MAPBOX_ACCESS_TOKEN')
import IPython.core.display as di
from demos.tools.go_utils import GoProject
# This line will hide code by default when the notebook is exported as HTML
di.display_html('<script>jQuery(function() {if (jQuery("body.notebook_app").length == 0) { jQuery("div.input").toggle(); jQuery(".prompt").toggle();}});</script>', raw=True)
# This line will add a button to toggle visibility of code blocks, for use with the HTML export version
di.display_html('''<button onclick="jQuery('div.input').toggle(); jQuery('.prompt').toggle();">Toggle code</button>''', raw=True)
di.Image('/home/jovyan/demos/assets/oi_header.png')
Out[20]:
Iran: Internet Shutdown Observed Using Geolocation Data¶
With Orbital Insight GO, users have access to a world of geospatial information (satellite, geolocation, etc). For any area of interest ("AOI"), users can count objects like cars & trucks in satellite imagery, monitor landuse change, and analyze mobile device traffic as a proxy for human foot traffic.
This brief demo shows how users can use GO to analyze geolocation (mobile device) data as another way of verifying internet shutdowns in closed-off countries.
Background: Internal Iranian Protests¶
Protests in Iran were initially started due to rises in gasoline prices. Demonstrators then increased their demands covering more political and regime related freedom issues. In an effort to scatter the protests, which allegedly killed 1,500 people within two weeks, the internet was shut down. Iranian officials and Iranian mobile operators denied there was an order to shut down the internet. The internet blockage therefore made it difficult for protestors to generate support and organize.
More information on the protest can be found here: https://www.reuters.com/article/us-iran-protests-internet/iran-curbs-internet-before-possible-new-protests-reports-idUSKBN1YT0GA
For Government Entities:¶
Monitoring the current geopolitical climate in countries that are relatively isolated, and not outwardly friendly to western countries is crucial for national security. A variety of sources are needed to corroborate reports, and better understand what is happening on the ground. HUMINT, SIGINT, etc., and now unclassified geolocation data can help verify specific happenings in these locations. This internet shutdown is one instance of acute government action taken against its citizens. Analysts can therefore set up GO projects to automatically monitor for anomalies which can corroborate other sources of data. Additionally, the geolocation data itself is unclassified, enabling more Department of Defense and Intelligence Community members to use GO that may only have a SECRET clearance instead of a TS/SCI.
Moving Forward:¶
A user interested in internet shutdowns can create projects and monitor for spikes or dips in geolocation data. Geolocation data, in this case, is an indication of internet being turned off by the government of Iran. Over time, a user can begin to correlate protests and other events with internet shutdowns and geolocation data. This historical data can eventually be used to help infer when future shut offs are going to happen as well as when an event is happening on the ground. Additionally, the internet may not be shutdown everywhere. By breaking up areas of interest into geohashes, or heatmaps, a user can more granularly identify if there are shut offs in some areas but not others. This information helps pinpoint where events are taking place, and guides analysts when adding more context to the activities on the ground.
Iranian Points of Interest¶
Viewing Project AOIs in GO Analysis¶
In [21]:
# This is the requisite Go project IDs for this use case demo
# Change this to your own project
project_id = 'G91G92jwZ4-200121'
In [22]:
# Instantiate completed GO Project
prj = GoProject(project_id)
In [23]:
# Gets DF of Target AOIs in a Project if needed
#target_aois = ['Tabriz','Tehran','Isfahan']
#gdf_aois = gdf_aois[gdf_aois['name'].isin(target_aois)]
# Get all project AOIs
gdf_aois = prj.get_aois()
gdf_aois.head()
Out[23]:
In [24]:
# Plot AOIs, hover cursor
# Calculate initial map center
centroids_x = gdf_aois['centroid_wkt'].apply(lambda g: g.x).mean()
centroids_y = gdf_aois['centroid_wkt'].apply(lambda g: g.y).mean()
geojson = gpd.GeoSeries(gdf_aois['boundary_wkt']).__geo_interface__
layers = []
layers.append(dict(type='fill',
line=dict(width=3.5),
opacity=0.8,
sourcetype='geojson',
source=geojson)
)
data = [
go.Scattermapbox(
lat=gdf_aois['centroid_wkt'].apply(lambda y:y.y),
lon=gdf_aois['centroid_wkt'].apply(lambda x:x.x),
mode='markers',
marker=go.scattermapbox.Marker(
size=14
),
text=gdf_aois['name']
)
]
fig = go.Figure(data=data)
fig.update_layout(
hovermode='closest',
height=500,
mapbox=go.layout.Mapbox(
accesstoken=mapbox_access_token,
bearing=0,
layers=layers,
center=go.layout.mapbox.Center(
lat=centroids_y,
lon=centroids_x,
),
pitch=0,
zoom=4.25
)
)
fig.show()
Charts and Heatmaps¶
In the following graphics, notice the decline in foot traffic from ~15 November to ~24 November, corroborating open source reports of internet shutdown. Zoom and pan within the interactive Plotly charts as needed.
In [25]:
# Call to view project parameters
prj.get_properties()
Out[25]:
In [26]:
# Pull foot traffic results
# Call to pull Foot Traffic results
# results = prj.get_foot_traffic_results()
results = prj.get_foot_traffic_results(sync=True, request=False)
df_raw_count = results.get_timeseries('raw.count')
df_raw_count.head()
# If created a normalized project
# df_uznorm_count = results.get_timeseries('uznorm.count')
# Print list of available timeseries (raw & normalized counts)
# print('List of available timeseries:', results.get_timeseries_list())
# Get normalized foot traffic timeseries ('uznorm.count') as a pandas DataFrame
# df_norm = results.get_timeseries('uznorm.count')
# Show preview of the normalized timeseries DataFrame
# display(df_norm)
Out[26]:
Embassies in Iran¶
In [27]:
# Function to visualize project results in Plotly
def plot_traffic(df_traffic, title, dropdown=True, start_dt=None, end_dt=None):
df = df_traffic.loc[start_dt:end_dt].copy()
data = []
buttons = []
for i, aoi in enumerate(df.columns):
# Chart data
trace = go.Scatter(
x = df.index,
y = df[aoi].values,
name = aoi,
visible = (i == 0) if dropdown else True
)
data.append(trace)
# Button
visible = [False] * len(df.columns)
visible[i] = True
button = dict(label=aoi, method='update', args=[{'visible': visible}])
buttons.append(button)
# Layout
layout = {
'title': dict(text=title, x=0.5),
'legend': dict(orientation='v'),
'height': 500
}
# Dropdown menu
if dropdown:
menu = go.layout.Updatemenu(
direction = 'down',
buttons = buttons,
showactive = True,
x=1.0,
xanchor='right',
y=1.22,
yanchor='top'
)
layout['updatemenus'] = [menu]
# Plot
fig = go.Figure(data=data, layout=layout)
iplot(fig)
In [28]:
# Daily time series of 3 specified AOIs:
df_embassy = df_raw_count[['UK_embassy','Iran_parliament','FR_embassy','RUS_embassy']]
# Plot results
title = 'Iran: Daily Raw UDC for Embassies'
plot_traffic(df_embassy, title)
Iranian Cities¶
In [29]:
# Daily time series of city AOIs:
df_city = df_raw_count[['Mashhad','Tabriz','Tehran','Isfahan']]
# Plot results
title = 'Iran: Daily Raw UDC for Cities'
plot_traffic(df_city, title, dropdown=False)
In [30]:
# To plot all time series on a single graph:
df_all = df_raw_count
title = 'Iran: Daily UDC for All Project AOIs'
plot_traffic(df_all,title, dropdown=False)
In [31]:
# Plot Weekly average values, within a specified time
df_roll = df_all.rolling(window=12).mean()
title = 'Iran: Weekly Mean Normalized of UDC of All Project AOIs'
plot_traffic(df_roll, title, dropdown=False,start_dt='2019-01-01', end_dt='2020-01-15')
Iranian Airfields¶
In [32]:
# This is the requisite Go project IDs for this use case demo
# Change this to your own project
project_id = 'L14fbh09Y5-200120'
In [33]:
# Instantiate completed GO Project
prj = GoProject(project_id)
In [34]:
# Gets DF of Target AOIs in a Project
#target_aois = ['Slaughter Lane','Brodie Lane','45 Toll']
gdf_aois = prj.get_aois()
#gdf_aois = gdf_aois[gdf_aois['name'].isin(target_aois)]
gdf_aois.head()
Out[34]:
In [35]:
# Plot AOIs, hover cursor
# Calculate initial map center
centroids_x = gdf_aois['centroid_wkt'].apply(lambda g: g.x).mean()
centroids_y = gdf_aois['centroid_wkt'].apply(lambda g: g.y).mean()
geojson = gpd.GeoSeries(gdf_aois['boundary_wkt']).__geo_interface__
layers = []
layers.append(dict(type='fill',
line=dict(width=3.5),
opacity=0.8,
sourcetype='geojson',
source=geojson)
)
data = [
go.Scattermapbox(
lat=gdf_aois['centroid_wkt'].apply(lambda y:y.y),
lon=gdf_aois['centroid_wkt'].apply(lambda x:x.x),
mode='markers',
marker=go.scattermapbox.Marker(
size=14
),
text=gdf_aois['name']
)
]
fig = go.Figure(data=data)
fig.update_layout(
hovermode='closest',
height=700,
mapbox=go.layout.Mapbox(
accesstoken=mapbox_access_token,
bearing=0,
layers=layers,
center=go.layout.mapbox.Center(
lat=centroids_y,
lon=centroids_x,
),
pitch=0,
zoom=4.25
)
)
fig.show()
View Project Parameters - Iranian Airfields¶
In [36]:
# Call to view project parameters
prj.get_properties()
Out[36]:
Pulling Project Results - Iranian Airfields¶
In [37]:
# Pull foot traffic results
# # Call to pull Foot Traffic results
#results = prj.get_foot_traffic_results()
results = prj.get_foot_traffic_results(sync=True, request=False)
df_raw_count = results.get_timeseries('raw.count')
df_raw_count.head()
# df_uznorm_count = results.get_timeseries('uznorm.count')
# Print list of available timeseries (raw & normalized counts)
#print('List of available timeseries:', results.get_timeseries_list())
# # Get normalized foot traffic timeseries ('uznorm.count') as a pandas DataFrame
# df_norm = results.get_timeseries('uznorm.count')
# # Show preview of the normalized timeseries DataFrame
# display(df_norm)
Out[37]:
Visualizing Project Results - Iranian Airfields¶
In [38]:
# Function to visualize project results in Plotly
def plot_traffic(df_traffic, title, dropdown=True, start_dt=None, end_dt=None):
df = df_traffic.loc[start_dt:end_dt].copy()
data = []
buttons = []
for i, aoi in enumerate(df.columns):
# Chart data
trace = go.Scatter(
x = df.index,
y = df[aoi].values,
name = aoi,
visible = (i == 0) if dropdown else True
)
data.append(trace)
# Button
visible = [False] * len(df.columns)
visible[i] = True
button = dict(label=aoi, method='update', args=[{'visible': visible}])
buttons.append(button)
# Layout
layout = {
'title': dict(text=title, x=0.5),
'legend': dict(orientation='v'),
'height': 500
}
# Dropdown menu
if dropdown:
menu = go.layout.Updatemenu(
direction = 'down',
buttons = buttons,
showactive = True,
x=1.0,
xanchor='right',
y=1.22,
yanchor='top'
)
layout['updatemenus'] = [menu]
# Plot
fig = go.Figure(data=data, layout=layout)
iplot(fig)
In [39]:
# Normalized daily time series of 3 specified AOIs:
df = df_raw_count[['Badr Airbase', 'Doshan Tappeh Airbase','Kerman Airport','Khatami Airbase' ]]
# Plot results
title = 'Daily Raw UDC, Single AOI'
plot_traffic(df, title)
In [40]:
# To plot all time series on a single graph:
df1 = df_raw_count
title = 'Iranian Airfields: Daily UDC, All AOIs'
plot_traffic(df1, title, dropdown=False)
In [41]:
# Plot Weekly average values, within a specified time
df1_roll = df1.rolling(window=5).mean()
title = 'Iran Airfields: Weekly Mean Normalized UDC, All AOIs'
plot_traffic(df1_roll, title, dropdown=False,start_dt='2019-11-01', end_dt='2020-01-15')
Heatmap Projects¶
Heatmaps are another way to show density of foot traffic over time in a more granular way. Using geohashes, we can see where in the AOI there are more or less pings. By adding a date slider grouped by week, we see that from ~ 15 November to 23 November there is relative foot traffic/mobile geolocation silence. This is also evident in the charts above when looking at AOI level geolocation data.
Tehran Heatmap GH5¶
In [42]:
# Creat heatmap project from foot traffic project and AOI
# aoi_name = 'Tehran'
# project_id = 'G91G92jwZ4-200121'
# geo_map.create_heatmap_proj(aoi=aoi_name,project_id=project_id,gh_length=5,heatmap_proj_name='Tehran',time_interval='DAY')
In [43]:
def plot_heatmap(heatmap_project_id,start_date,end_date,frequency,heatmap_title):
heatmap_proj = go_utils.GoProject(heatmap_project_id)
heatmap_results = heatmap_proj.get_foot_traffic_results()
result_df = heatmap_results.get_timeseries('raw.count', form='long')
result_df.rename(columns={'aoi_name':'geohash'}, inplace=True)
# group data by specified frequency
result_df = result_df.groupby(['geohash', pd.Grouper(key='obs_date', freq=frequency)], as_index=False).agg({'unique_count': 'sum','obs_date':'last','timeseries':'last'})
result_df = pd.DataFrame(result_df)
start_date_dt = datetime.datetime.strptime(start_date,'%Y-%m-%d')
end_date_dt = datetime.datetime.strptime(end_date,'%Y-%m-%d')
date_range = pd.date_range(start_date_dt,end_date_dt,freq=frequency)
result_df = result_df[(result_df['obs_date'] >= start_date) & (result_df['obs_date'] <= end_date)]
data_df = pd.DataFrame(result_df)
data_df['geometry'] = result_df['geohash'].apply(geohash_to_polygon)
data_gdf = gpd.GeoDataFrame(data_df)
num_days = len(date_range)
sliders = []
steps = []
data = []
# Loop through dates to add traces to data
for i, date in enumerate(date_range):
temp_df = gpd.GeoDataFrame(data_gdf[data_gdf['obs_date']==pd.to_datetime(date)])
temp_df.drop(columns=['obs_date'],inplace=True)
locations = temp_df.index
text=temp_df['unique_count']
geojson = json.loads(temp_df.to_json())
data.append(go.Choroplethmapbox(geojson=geojson,
locations=locations,
visible=i==0,
hoverinfo='text',
text=text,
z=temp_df['unique_count'],
marker=dict(opacity=.5,line=dict(color='#ffffff')),
zmin=data_gdf['unique_count'].quantile(.05),
zmax=data_gdf['unique_count'].quantile(.95),
colorscale='plasma'))
visible = [False] * num_days
visible[i] = True
arg_dict = {'visible':visible}
step = dict(method='update',
args=[arg_dict],
label=date.strftime('%Y-%m-%d'))
steps.append(step)
# Add day slider
sliders.append(dict(active=0,
currentvalue={'prefix':'Date: '},
pad={'t':125},
steps=steps,
len=1,
ticklen=0,
tickwidth=0,
minorticklen=0))
# Add opacity slider
steps=[]
for ix in range(0,101):
arg_dict={}
arg_dict['marker.opacity'] = float(ix)/100.0
step = dict(method='restyle',
args=[arg_dict],
label='{:.0f}%'.format(ix))
steps.append(step)
sliders.append(dict(active=50,
currentvalue={'prefix':'Opacity: '},
pad={'t':35,'b':5},
steps=steps,
len=.45,
ticklen=0,
tickwidth=0,
minorticklen=0))
# Get centroid for map
centroid_y = sum([geom.centroid.y for geom in data_gdf['geometry']])/len(data_gdf)
centroid_x = sum([geom.centroid.x for geom in data_gdf['geometry']])/len(data_gdf)
# Set layout properties
layout = go.Layout(title=heatmap_title,
height=700,
width=950,
sliders=sliders,
mapbox=dict(accesstoken=mapbox_access_token,
bearing=0,
center=dict(lat=centroid_y,
lon=centroid_x),
pitch=0,
style='light',
zoom=12))
fig=go.Figure(data=data,layout=layout)
iplot(fig)
In [44]:
heatmap_project_id = 'p8EYmHtumE-200123'
start_date = '2019-10-01'
end_date = '2020-01-01'
frequency = 'w' # w for week, m for month
heatmap_title = 'Tehran Weekly Foot Traffic'
plot_heatmap(heatmap_project_id, start_date, end_date, frequency, heatmap_title)
Tabriz Heatmap GH5¶
In [45]:
# Creat heatmap project from foot traffic project and AOI
# aoi_name = 'Tabriz'
# project_id = 'G91G92jwZ4-200121'
# geo_map.create_heatmap_proj(aoi=aoi_name,project_id=project_id,gh_length=5,heatmap_proj_name='Tabriz',time_interval='DAY')
In [46]:
heatmap_project_id = 'L9Vra2USFn-200123'
start_date = '2019-10-01'
end_date = '2020-01-01'
frequency = 'w' # w for week, m for month
heatmap_title = 'Tabriz Weekly Foot Traffic'
plot_heatmap(heatmap_project_id, start_date, end_date, frequency, heatmap_title)
Mehrabad International Airport Heatmap GH7¶
In [47]:
heatmap_project_id = 'k2W5cjZ8yl-200123'
start_date = '2019-10-01'
end_date = '2020-01-20'
frequency = 'w' # w for week, m for month
heatmap_title = 'Mehrabad International Airport Weekly Foot Traffic'
plot_heatmap(heatmap_project_id, start_date, end_date, frequency, heatmap_title)
Conclusion¶
Based off of the foot traffic data at certain locations and timeframes, we can assume that:
- The internet was in fact shut down from ~ 15 November 2019 - 24 November 2019.
- This shutdown corresponds with internal protest timelines.
- Near non-existant and/or severe declines in geolocation data is seen in relatively small AOIs such as embassies and airports/airfields.
- Additionally, severe geolocation declines are also seen at the city level.
In [ ]: